@fengelniederhammer thanks! Is there anything that might break stuff for us? Can't see anything immediately looking at changelog, I assume you would have told us if there was something breaking, but better check explicitly with you :)

We should play with TSV downloads and see if our new order looks reasonable as a starting point (doesn't need to be perfect - just not really bad!)

There should be nothing that is relevant for Loculus, except for the "order CSV/TSV columns" feature in LAPIS.

The changes in SILO were mostly internal stuff or preprocessing issues that we had with large datasets.

Is there anything we could/need to do to take advantage of new features? In particular, from the changelog I'm wondering about the following:

• lapis: add healthcheck to Docker containers (0944990), closes Healthcheck GenSpectrum/LAPIS#813
• lapis: order CSV/TSV columns of aggregated/details data as specified in the request fields or as in the database config (6cf8704), closes Order TSV and CSV columns as defined in config and fields GenSpectrum/LAPIS#910
• make duckdb memory limit configurable via preprocessing config (c112eb1)

Btw, great changelog, well done @fengelniederhammer and @Taepper on this!

We need to make sure we test this well, e.g. on staging, before merging as in the past there've been unexpected change in SILO/LAPIS. We should probably come up with a checklist for things to do whenever we upgrade SILO/LAPIS versions, including integrity checks etc.

Here is a start for the checklist, feel free to add more points (added to main PR description):

• lapis: add healthcheck to Docker containers

Loculus could benefit, but AFAIK the LAPIS deployments already have there custom healthchecks implemented in the Helm chart. There would not be significant improvement.

• make duckdb memory limit configurable via preprocessing config

Given the current size of datasets, this should be pretty much irrelevant. We were having issues because our datasets are in the order of magnitude of 100 GB.

lapis: add healthcheck to Docker containers

Loculus could benefit, but AFAIK the LAPIS deployments already have there custom healthchecks implemented in the Helm chart. There would not be significant improvement.

It's best to push anything to dependency if possible and only redo what's necessary. So I would welcome if you showed us how to make use of the new health check to simplify/minifiy our helm config. Not blocking, we can make an issue for this as well.

How do we move forward on this?

* [ ]  Manual testing:
  
  * [ ]  Let cook on pathoplexus staging to see if all is good before merging here
  * [ ]  Integrity check on pathoplexus staging vs prod, using snakemake test pipeline

I don't have access to Pathoplexus deployment infrastructure, someone else would need to deploy and test.
(Also I see quite low risk in updating to these versions as there have been very few changes with quite little impact - of course still good to test.)

* [ ]  Adjust configs to take advantage of new features
  * [ ]  Health checks

I don't see much benefit from implementing the new health check.

  * [ ]  TSV column order

That should be it's own feature - implemented in a separate issue.

I don't have access to Pathoplexus deployment infrastructure

You likely do have access like anyone else: by making PRs - but it's true that it's easier for me to do it as I know exactly which PRs to make.

But can you still check if you have permission to make such PRs? Would not harm if you knew how the deployment works, just in case!

It's quite easy:

• Make PR in pathoplexus, pointing at a loculus commit you want to test: Test https://github.com/loculus-project/loculus/pull/2934 pathoplexus/pathoplexus#218
• Point staging sha at the pathoplexus commit you want to test in staging: Test https://github.com/loculus-project/loculus/pull/2934 pathoplexus/loculus_deployments#108

I've now put it on staging, but the PRs above show how to do it :)

(If Fabian had made a PR it also wouldn't have triggered an image build so would require a bit of additional work to deploy - just for info, not arguing one way or the other)

(If Fabian had made a PR it also wouldn't have triggered an image build so would require a bit of additional work to deploy - just for info, not arguing one way or the other)

Thanks I wasn't sure about that - would just require triggering an action run, but as approval is necessary anyways this is not a real blocker either.

What's actually the benefit of testing on Pathoplexus? Why isn't testing on the Loculus preview enough?

Just that the fields may be in a different order in PP - though on reflection they are probably not! Checking that the order in yaml is the same and testing the loculus preview would also work.

Had a look at the preview

We should probably move accessionVersion to be next to accession and version on pathoplexus alongside merging this. Also the dataUseTermsUrl is split apart from the other data use terms.

I wonder whether we want to move versionStatus and versionComment next to version?

What's actually the benefit of testing on Pathoplexus? Why isn't testing on the Loculus preview enough?

Loculus has only artificial data, lacks real users, real user submitted data etc - lots of things we can catch on PP that we can't on Loculus.

@anna-parker and I tested on staging:

• Fields in tsv are ordered differently (as expected), but no value changes as verified with csv-diff
• We tested that revoking makes it through as well as a manual test just because we can now test this